fix(eval): calibrate canary latency gates to the runner's measurement context#624
Conversation
… context Issue #459 residual: after #606 every quality metric is perfect (grounded supported 1.0, citation/numeric failures 0) and the only red gates are p95_latency_ms 48256 > 25000 and route extractive 48256 > 12000. Those thresholds were calibrated when fast confidence-gate refusals dominated the sample; real grounded answers measured from cross-region GitHub runners pay full generation time, and the provider-timeout -> extractive-fallback chain costs 30s+ per affected case. Raise the overall and extractive-route gates to 60s for this measurement context. User-facing latency stays enforced by the answer SLO deep probe, which is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe evaluation script now applies higher RAG latency thresholds when running in the cross-region context. The canary workflow sets this context for its live-generation quality step. ChangesEvaluation latency context
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91363dec39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. After fixing or dispositioning a thread, reply in that thread with as the first line, followed by a concise summary; that marker authorizes the workflow to close that exact thread. If human input or new authorization is required, do not use the marker and leave the thread open with the blocker. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
Codex review (P2): the wider gates leaked into every eval-quality caller, including eval:quality:release, quietly relaxing the documented 25s/12s release ceilings. The strict values are the defaults again; the Eval Canary workflow opts into the cross-region allowance explicitly via EVAL_LATENCY_CONTEXT=cross-region-runner on the answer-quality step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e45aa73f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ragRouteP95LatencyMs: { | ||
| unsupported: 4_000, | ||
| extractive: 12_000, | ||
| extractive: crossRegionRunnerLatencyContext ? 60_000 : 12_000, |
There was a problem hiding this comment.
When EVAL_LATENCY_CONTEXT=cross-region-runner and a generation attempt times out but recovers with an extractive fallback, this route ceiling is not the one the eval applies: latencyRouteForAnswer charges any answer with generation_latency_ms > 0 to the attempted fast/strong route, and the existing eval-quality test explicitly covers an extractive fallback reported as fast. Because only the extractive ceiling is widened here, a 30s+ provider-timeout fallback can still fail the canary under the unchanged 25s/35s attempted-route gates, which leaves the cited failure mode unfixed for cross-region runs.
Useful? React with 👍 / 👎.
1 participant
Summary
Closes the #459 residual. After #606 every canary quality metric is perfect (grounded supported 1.0, citation/numeric failures 0, expected-source hit 1.0); the only red gates are
p95_latency_ms 48256 > 25000androute extractive 48256 > 12000. Those thresholds were calibrated when fast confidence-gate refusals dominated the sample — real grounded answers measured from cross-region GitHub runners pay full generation time, and the provider-timeout→extractive-fallback chain costs 30s+ per affected case. This raises the overall and extractive-route canary gates to 60s for that measurement context, per the option 1 recommendation already triaged on the issue.Not touched: the answer SLO deep probe (
/api/health?deep=1), which is what enforces user-facing latency;fast/strong/unsupportedroute gates (they did not fail and stay as regression tripwires).Verification
format:check,lint,typecheckon freshnpm ci🤖 Generated with Claude Code
Summary by CodeRabbit